Thermopile Array Fusion Tracking

ABSTRACT

A simultaneous location and mapping (SLAM)-enabled video game system, a user device of the video game system, and a computer-readable storage medium of the user device are disclosed. Generally, the video game system includes a video game console, a plurality of thermal beacons, and a user device communicatively coupled with the video game console. The user device includes a thermopile array, a processor, and a memory. The user device may receive thermal data from the thermopile array, the thermal data corresponding to a thermal signal emitted from a thermal beacon of the plurality of thermal beacons and detected by the thermopile array. The user device may determine, based on the thermal data, its location in 3D space, and then transmit that location to the video game system.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/297,292, filed Mar. 8, 2019, which is hereby incorporated byreference in its entirety for all purposes.

BACKGROUND

Simultaneous localization and mapping (SLAM) is a technique whereby amap is constructed of a particular environment while simultaneouslykeeping track of a target object's location within the environment. SLAMis used in robotic mapping and navigation in an open environment,including self-driving cars. SLAM may also be used in a closedenvironment, such as a room for playing video games or conducting avirtual meeting.

Existing SLAM techniques often use some type of camera configurationthat contains one or more cameras, each camera containing one or moreoptical sensors. A variety of optical sensor types may be used,including one-dimensional (1D), 2D, 3D, etc. For example, in a casewhere two fixed-point 2D cameras are employed, each camera may detect atarget object. The SLAM application's algorithm may determine thedistance between the target object and each camera and triangulate thisinformation to determine the location (e.g., position) of the targetobject. Often, especially, when the target may be moving and/or rotatingthrough space, an inertial measurement unit (IMU) may also be employed,along with a camera apparatus, to detect the target's linearacceleration and rotation rate. The combination of sensor input fromboth an IMU and a camera apparatus helps to enable a higher degree oflocation-tracking accuracy. Other SLAM techniques also exist, includingradar SLAM, WiFi-SLAM, etc.

However, there are at least two problems with existing SLAM-basedapplications. First, when employing an IMU to track the acceleration androtation of a moving target through space, the IMU location-trackingaccuracy degrades continuously over time due at least to offset error(also known as “drift”), thus decreasing the accuracy of SLAMtechniques. Second, while combining sensory data received from otherequipment (e.g., cameras) can help improve the SLAM algorithm'saccuracy, this equipment currently can add significant cost and requirea complex setup procedure. Therefore, there is a need to improveexisting SLAM techniques by providing a lower-cost and yet highlyaccurate location-tracking solution.

BRIEF SUMMARY

Generally, techniques for determining user device location aredescribed. In an example, a video game system includes a video gameconsole, a plurality of thermal beacons, and a user devicecommunicatively coupled with the video game console. The user deviceincludes a thermopile array. The user device further includes aprocessor and a memory that stores instructions that, upon execution bythe processor, cause the processor to perform operations. In oneoperation, the processor receives thermal data from the thermopilearray. The thermal data corresponds to a thermal signal emitted from athermal beacon of the plurality of thermal beacons and detected by thethermopile array. In another operation, the processor determines, basedon the thermal data, a location of the user device in athree-dimensional (3D) space. In another operation, the processortransmits the location of the user device to the video game console.

In an example, the user device may also include an IMU. The processor ofthe user device may further perform operations. In one operation, theprocessor receives IMU data from the IMU. The IMU data includesacceleration data corresponding to an acceleration of the user device inthe 3D space and orientation data corresponding to a rotational rate ofthe user device in the 3D space. In another operation, the processordetermines the location of the user device in the 3D space by inputting,to a sensor fusion algorithm, the thermal data, the IMU data, and aprevious location data. In one example, the processor determines aninitial location of the user device based on the thermal data andindependent of the IMU data of the IMU. In another example, the previouslocation data is stored on the user device. In yet another example, thesensor fusion algorithm utilizes an artificial intelligence modeltrained to determine the location. In another example, the sensor fusionalgorithm utilizes a Kalman filter.

In an example, the sensor fusion algorithm generates a confidence valuecorresponding to the thermal data. The confidence value is based on anumber of thermal beacons of the plurality of thermal beacons in a fieldof view of the thermopile array. The confidence value is used by thesensor fusion algorithm to determine the location of the user device.

In an example, the processor of the user device may further performoperations. In one operation, the processor receives second thermal datafrom the thermopile array. The second thermal data corresponds to asecond thermal signal emitted from a second thermal beacon of theplurality of thermal beacons and detected by the thermopile array. Thethermal beacon and the second thermal beacon are simultaneously in afield of view of the thermopile array. The location of the user deviceis further determined based on the second thermal data.

In an example, the thermal signal emitted by a thermal beacon of theplurality of thermal beacons includes an identifier unique to thethermal beacon. The identifier is used by the thermopile array toidentify the thermal beacon from the plurality of thermal beacons.

In an example, each thermal beacon of the plurality of thermal beaconsis positioned in a gaming environment. The 3D space is mapped to aportion of the gaming environment, and the thermal beacons arepositioned such that, for a particular position of the user devicewithin the 3D space, the thermopile array can detect thermal signalsfrom at least two thermal beacons.

In an example, the user device is a video game controller. In anotherexample, each beacon of the plurality of thermal beacons is an infrareddiode.

A user device is also described. The user device includes a thermopilearray. The user device further includes a processor and a memory thatstores instructions that, upon execution by the processor, cause theprocessor to perform operations which are disclosed herein above.

A non-transitory computer-readable storage medium that storesinstructions is also described. The instructions, upon execution on auser device, configure the user device to perform operations disclosedherein above.

In an example, the non-transitory computer-readable storage medium isfurther configured to perform calibration operations. In one operation,the user device determines a first location of the user device based onfirst thermal data corresponding to first thermal signals emitted from afirst set of thermal beacons of the plurality of thermal beacons thatare positioned in the 3D space. In another operation, the user devicedetermines a second location of the user device based on second thermaldata corresponding to second thermal signals emitted from a second setof thermal beacons of the plurality of thermal beacons that arepositioned in the 3D space. In another operation, the user devicesgenerates, based on the first location and the second location, a 3Dmodel of the 3D space. In yet another operation, the user device storesthe 3D model.

In an example, the non-transitory computer-readable storage medium isconfigured to perform further calibration operations. In one operation,the user device receives an instruction that requests a user to move theuser device to the first location. In another instruction, responsive tothe determining the first location, the user device receives aninstruction that requests the user to move the user device to the secondlocation.

Some embodiments of the present disclosure provide several technicaladvantages over current techniques for determining user device location.First, the present disclosure provides a method for achieving similarlocation-tracking accuracy compared to existing SLAM techniques whilesignificantly reducing financial and human resource costs. For example,the cost of thermal beacons (e.g., infrared diodes) is inexpensive andeasy to affix onto the walls of a room. Similarly, a thermal sensor isalso inexpensive, compared to the cost of one or more optical sensors(e.g. components of a camera). Additionally, the present disclosure maynot only be used as an alternative to existing SLAM techniques, but maybe used to improve the accuracy of location-tracking in existing systemsand methods.

A further understanding of the nature and the advantages of theembodiments disclosed and suggested herein may be realized by referenceto the remaining portions of the specification and the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a system using a thermopile array toimplement SLAM application, according to an embodiment of the presentdisclosure.

FIG. 2 illustrates a user device that includes a thermopile array whichreceives signals from one or more thermal beacons, according to anembodiment of the present disclosure.

FIG. 3 is a block diagram of an example architecture for a user deviceutilizing a thermopile array to implement SLAM, according to anembodiment of the present disclosure.

FIG. 4 illustrates an example flow for performing calibration of theuser device, according to embodiments of the present disclosure.

FIG. 5 illustrates an example flow for implementing SLAM on a userdevice that includes a thermopile array, according to embodiments of thepresent disclosure.

FIG. 6 illustrates an example flow for implementing SLAM on a userdevice that includes a thermopile array and an IMU, according toembodiments of the present disclosure.

FIG. 7 illustrates an example of a hardware system suitable forimplementing a computer system, according to embodiments of the presentdisclosure.

DETAILED DESCRIPTION

Generally, systems and methods for utilizing a thermal sensor array todetermine a user device location are described. Typically, a user mayoperate a user device (e.g., a video game controller, headset, paddle,etc.) within a 3D space, the 3D space being mapped to a portion of aphysical space (e.g., a video game room). The user may interact with theuser device by moving it in different directions and at different speedswithin the 3D space. The user device (e.g., a video game controller, avirtual reality (VR) headset, etc.) will also typically interact with acomputing hub (e.g., a video game console), which may in turn interactwith other devices and cause them to perform a function (e.g., change avideo image being displayed on a television (TV)). One of theinteractions the user device may have with the computing hub is todetermine its location within the 3D space and send the location to thecomputing hub. The user device may determine its location by utilizing athermal sensor array of the user device. The thermal sensor array may beconfigured to receive thermal signals from one or more thermal beacons(e.g., IR LEDs), which are positioned in the physical space (e.g.,affixed to the walls of the game room). Based on thermal data receivedfrom the thermal sensor array, the user device may determine itslocation in the 3D space, and transmit that location to the computinghub. The user device may also use the thermal data, together with othersensor data to determine its location.

In an example, the user device may also include and receive data from anIMU. The user device may also have stored a previous location (e.g.,last determined location). In addition to the thermal data received fromthe thermal sensor array, the user device may input the IMU data andprevious location data into a fusion sensor algorithm. The fusion sensoralgorithm may use the different data inputs to determine the user devicelocation. The user device may then store that location data to be usedin the future, for example, as a previous location data.

The above examples are provided for illustrative purposes. Theembodiments of the present disclosure are not limited as such. Theembodiments similarly apply to a larger number of SLAM applicationsusing thermal sensor arrays to determine location. These and otherembodiments are further described herein next.

FIG. 1 illustrates an example of a system 100, which includes a userdevice that further includes a thermal sensor array for use indetermining a location of the user device, according to an embodiment ofthe present disclosure. In an example, the location determinationimplements a SLAM application. In FIG. 1, a gaming room 126 is depicted,wherein a user 102 wears a gaming headset 104 (e.g., the user device) tointeract with a video game console 106. The video game console in turnmay cause, for example, a TV 108 to display movement of an object on thescreen which corresponds to the gaming headset's 104 movement. It shouldbe understood that although FIG. 1 and subsequent illustrations maydepict a SLAM application within a gaming environment, the use of thistype of scenario should not be construed to pose a limitation on thescope of the disclosure. For example, in some embodiments, a physicalspace such as a meeting room or theater space may be used and otherlocation determination applications are possible. In general, andcontinuing with the gaming room example, the gaming room 126 may allowfor a plurality of thermal beacons 110-124 (e.g., IR LEDs) to bepositioned in the room 126.

The user device 104 should include one or more thermal sensors (e.g.,thermal sensor array or thermopile array) that are configured to receivethermal signals from one or more thermal beacons 110-124 positioned inthe room 126. It should be understood that, although a thermopile arrayis type of thermal sensor that is depicted in FIG. 1 and subsequentfigures, other types of thermal sensors may be used as a suitablethermal sensor. Furthermore, although gaming headset 104 is a type ofuser device that is depicted below, other types of mobile computingdevices that include thermal sensors may be used as a suitable userdevice. This may include, but is not limited to, a video gamecontroller, a gaming paddle, a mobile phone, a laptop, etc.

In some embodiments, each beacon of the plurality of thermal beacons110-124 positioned in the room 126 may emit a thermal signal (e.g.,infrared light). Optionally, each thermal signal emitted by a thermalbeacon may include an identifier that is unique to the particularthermal beacon. For example, the identifier may be a modulated frequencyof the infrared signal, which the thermopile array of user device 104may be able to detect as a unique signal that corresponds to aparticular thermal beacon in the room. As discussed in more detailbelow, in some embodiments, user device 104 may be able to improve itslocation-tracking accuracy by triangulating its position relative to atleast two thermal beacons in the room 126.

In some embodiments, the plurality of thermal beacons 110-124 may bepositioned in the room 126 such that, for a particular position of theuser 102 within the 3D space of the room 126, the thermopile array ofthe user device 104 may be able to detect thermal signals from at leasttwo thermal beacons of the plurality within the thermopile array's fieldof view. For example, as depicted in FIG. 1, thermal beacons 110-124 mayeach be affixed to a wall of the gaming room 126, and be substantiallyequally spaced from each other. During a calibration process (discussedin detail below), the user 102 may be instructed to move the user device104 in an arc-like (e.g., 360 degree) motion. This calibration processmay allow the user device 104 to construct and store on the user device104 a 3D model of the 3D space, the 3D space corresponding to at least aportion of the room 126, and in which the user 102 may operate the userdevice 104 within. The calibration process may also verify that thethermopile array's field of view can detect at least two signals from aparticular point within the 3D space in the room 126. In otherembodiments, and depending on the type of SLAM application (e.g., therange of use intended for the user device within the room 126), two ormore thermal beacons may be positioned in the physical space 126 inorder for the user device 104 to determine its location.

FIG. 2 illustrates an example close-up view of a user device 200 whichreceives thermal signals from one or more thermal beacons. The userdevice 200 is a gaming headset and may correspond to the user device 104of FIG. 1. In an example, the headset 200 is a virtual reality (VR) oraugmented reality (AR) headset. Generally, the headset 200 includes ahousing 210 that may integrate components such as a display, processingunits, memories, audio systems, I/O ports, graphics processing units,network communications devices, and other electronic, electrical, andmechanical components. The housing 210 further integrates (e.g., houses,attaches, or holds) additional components for location tracking, suchthat the additional components are rigidly connected with the housing210. These components include, for instance, an IMU 214 and a thermopilearray 218.

In an example, the IMU 214 includes accelerometer(s), gyroscope(s), andmagnetometer(s). An accelerometer(s) measures movement along the X, Y,and Z axes. Generally, a gyroscope(s) measures 360 degree rotation. Amagnetometer(s) determines orientation towards a magnetic field. Assuch, inertial data (e.g., including acceleration data, orientationdata, and/or rotation data) indicative of a rotational motion of theheadset 200 can be generated from readings of these sensors.Translational motion can also be generated based on speed and time ofthe user's 102 head motion via the headset 200. For instance, a motionvector is defined. Speed is measured from the acceleration and distanceis measured as a function of speed and time. Direction is derived fromthe rotation and orientation. The motion vector defines the distance anddirection of the motion, thereby allowing a tracking of thetranslational motion of the headset 200 along the X, Y, and Z axes.Thus, by defining inertial data, distance, and direction in a motionvector, the motion of the user's headset 200 can be tracked over time.Accordingly, based on the inertial data, and by performingintegration(s) with respect to time, the location of headset 200 may bedetermined for a specific point in time in 3D space. A processing unitof the IMU 214 (e.g., a signal processor) may generate location datafrom data sensed by such IMU sensors.

As described above, one of the limitations with utilizing an IMU todetermine location is that an IMU typically suffers from accumulatederror. Because a SLAM application may be continually integratingacceleration with respect to time to calculate velocity and position,any measurement errors, however small, are accumulated over time,leading to drift. Drift is measurement of the difference between where auser device may initially determine it is located compared to its actuallocation. As such, the headset 200 also may contain a thermopile array218, which the headset 200 can use to continually correct for drifterrors and determine a more accurate location.

In an example, the thermopile array 218 is thermal infrared sensor arraythat is configured to measure temperature of one or more thermal beacons204, 206 from a distance by detecting the infrared energy from the oneor more thermal beacons 204, 206 (which may correspond to one or more ofthe thermal beacons 110-124 of FIG. 1). The thermopile array may includethermocouples that are connected on a silicon chip, which are configuredto convert thermal energy received from a thermal signal of a thermalbeacon 204, 206 into electrical energy in the form of a voltage output.Typically, the input thermal energy is proportional to the voltageoutput. Thus, because a user device 200 may detect more light energy(e.g., signal) the closer it is to a thermal beacon (e.g., creating ahigher temperature differential), a thermopile array can be used tomeasure the distance between a user device 200 and a particular thermalbeacon. In some embodiments, a processing unit connected to thethermopile array 218 (e.g., a signal processor which be integrated withthe thermopile array 218) may transmit thermal data, corresponding todistance from the user device 200 to the thermal beacon 204, 206, basedon the voltage output by an element of the thermopile array 218. In someembodiments, the thermopile may be composed of a single element (e.g.,pixel), dual element, etc. In other embodiments, the thermopile may be alinear (e.g., 16, 32) or area (e.g., 32×32) array of pixels. Althoughthe embodiments discussed herein discuss a thermopile array, thisdisclosure should not be construed to be so limiting. In someembodiments, any given pixel(s) of the thermopile array 218 may be ableto detect thermal signals from one or more thermal beacons at a giventime, provided that the thermal beacon is within the pixel's field ofview. As discussed above, and further below in regards to FIG. 3, thethermal data generated from the thermopile array 218 may be used totriangulate the distance between the user device 200 and at least twothermal beacons 204, 206 to determine a 3D location of the user device200 in a physical space 126. In some embodiments, the thermal data fromthe thermopile array may also be combined with the IMU data to output a3D location of the user device 200 in a physical space 126.

FIG. 3 is a block diagram 300 of an example architecture for a userdevice 302 (which may correspond to user device 104 of FIG. 1 and/oruser device 200 of FIG. 2) and utilizing a thermopile array to implementSLAM, according to an embodiment of the present disclosure. The userdevice 302 may include at least one memory 304, one or more processingunits (or processor(s)) 316, a thermopile array 318, an IMU 320, and acommunications device 322, among other components. The processor(s) 316may be implemented as appropriate in hardware. The thermopile array 318of the user device 302 may be configured to detect one or more thermalsignals (e.g., infrared light) from one or more thermal beacons 204,206. The communications device 322 may further be configured tocommunicate with a computing hub 106 (e.g., a video game console,virtual meeting server, etc.) using any suitable communication path.This may include, for instance, a wire or cable, fiber optics, atelephone line, a cellular link, a radio frequency (RF) link, a WAN orLAN network, the Internet, or any other suitable medium.

The memory 304 may store program instructions that are loadable andexecutable on the processor(s) 316, as well as data generated during theexecution of these programs. Depending on the configuration and type ofuser device 302, the memory 304 may be volatile (such as random accessmemory (RAM)) and/or non-volatile (such as read-only memory (ROM), flashmemory, etc.). In some implementations, the memory 304 may includemultiple different types of memory, such as static random access memory(SRAM), dynamic random access memory (DRAM) or ROM. The user device 302may also include additional storage (not shown), such as eitherremovable storage or non-removable storage including, but not limitedto, magnetic storage, optical disks, and/or tape storage. The diskdrives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for the computing devices.

Turning to the contents of the memory 304 in more detail, the memory 304may include an operating system 306 and one or more application modulesor services for implementing the features disclosed herein, including acalibration module 308, a sensor fusion algorithm module 312, and athermopile array algorithm module 314. It should be understood that anyof the tasks performed by one module may be performed by one or more ofthe other modules, and, as such, the module definitions provided hereinare included for illustrative purposes.

The operating system 306 may provide executable program instructions forthe general administration and operation of user device 302 andtypically will include a computer-readable storage medium (e.g., a harddisk, random access memory, read only memory, etc.) storing instructionsthat, when executed by a processor of the user device 302, allow theuser device 302 to perform its intended functions. Suitableimplementations for the operating system are known or commerciallyavailable and are readily implemented by persons having ordinary skillin the art, particularly in light of the disclosure herein.

The calibration module 308 may be responsible for determining andmaintaining in memory 304, a 3D model of a 3D space, wherein the 3Dspace may correspond to at least a portion of a physical space (e.g.,gaming room 126). In some embodiments, the calibration module 308 mayalso be responsible for verifying that the thermopile array 318 of theuser device 302, for a given position within the 3D space, can detect athermal signal from at least two beacons of the plurality of beacons110-124 (e.g., the beacons are within the field of view of one or moreelements of the thermopile array). This helps to enable triangulation oflocation via the thermopile array algorithm module 314, discussed below.In some embodiments, the calibration module 308 may perform operationsas described in FIG. 4. In one embodiment, and using FIG. 1's gamingroom 126 as an example, the 3D model (which may be represented as a 3Dwireframe model) corresponds to the 3D space in which the user 102expects to operate the user device 104 within. For example, the user mayonly intend to operate the user device 104 within a portion of thegaming room near the center (and radiating out from the center somedistance that is reachable by a human). The calibration module 308 mayinstruct the user to begin the calibration by moving the device to thecenter of the room 128 (e.g., where they primarily intend to operate thedevice) and then, while standing and turning in place, move the device104 in a 360 degree visual sweep of the room, divided into smaller arcsegments (e.g., 90 degree turns). At each segment, as described above,the user device 104 may verify at that at least two thermal beacons arewithin the field of view of the thermopile. Additionally, the userdevice 104 may determine and store in memory 304 the distance betweenthe user device 104 and each thermal beacon 204, 206 visible within thearc segment. Once the user device 104 completes the sweep of the room,the calibration module 308 may use the location data determined betweenthe user device 104 and each of the beacons 110-124 (relative to thecenter 128 of the room) to construct a 3D model of the 3D space. In someembodiments, the origin of the X, Y, and Z axes may be at the position(e.g., center 128) where the user 102 initially positioned the userdevice 104 during calibration. In some embodiments, a full 360 degreevisual sweep may not be needed to perform calibration. For example, ifthe user 104 will typically be operating the user device 104 facingforward towards the TV 108, the calibration may only recommend a partial(e.g., 180 degree) sweep of the room. In that case, a smaller number ofthermal beacons may be needed to pre-position in the room in order toconstruct the 3D model.

The sensor fusion algorithm module 312 may be responsible fordetermining a location of the user device 302. In some embodiments, thesensor fusion module 312 may be performed after calibration 308 has beenperformed. The sensor fusion module 312 may combine sensory data inputfrom one or more sources to improve accuracy in determining the locationof the user device 302. In an embodiment, the sensor fusion module 312may receive thermal data as sensory input from the thermopile array 318and IMU data as sensory input from the IMU 320.

In some embodiments, before combining the sensory input data from thedifferent sensors, the sensor fusion module 312 may first execute athermopile array algorithm module 314. The thermopile array algorithmmodule 314 may be responsible for triangulating, based on two or moredistance values (e.g., corresponding to the distances between the userdevice 302 and at least two thermal beacons 204, 206), the location ofthe user device 302. The location (e.g., position) may be in the form ofX, Y, and Z coordinates within the 3D model determined duringcalibration. In some embodiments, the thermopile array algorithm module314 may, in addition to outputting a location, also output a confidencevalue that corresponds with the location. The confidence value mayincrease or decrease depending on the number of thermal beacons detectedwithin the thermopile array's field of view (e.g., the number ofdistance values obtained). For example, if only one thermal beacon isdetected, although the thermopile array algorithm module 314 may stilloutput a location value, the corresponding confidence value may be low.Conversely, if two or more thermal beacons are detected within the fieldof view, the module 314 may output a high confidence value. In someembodiments, the user device 302 may execute the thermopile arrayalgorithm module 314 on a certain frequency. For example, the module 314may run at 120 Hz (approximately every 8.33 milliseconds). Typically, ahigher frequency of generating an updated location based on the thermaldata will improve location-tracking accuracy. For example, when fusingboth thermal data and IMU data (discussed further below), more frequentlocation information based on thermal data will help to correct againstdrift error within the IMU data.

In another embodiment, and returning to the sensor fusion module 312discussed above, the module 312 may receive location information (X, Y,and Z coordinates) from the thermopile array algorithm module 314 (e.g.,after the algorithm 314 used the thermal data to determine a 3D locationvalue and a corresponding confidence value) as well as locationinformation (X, Y, and Z coordinates) from the IMU data from the IMU320. The sensor fusion module 312 may combine (or “fuse”) the differentlocation data together using one or more of a number of algorithms andoutput a single location in 3D space with higher accuracy.

In one embodiment, the sensor fusion module 312 may employ an artificialintelligence model that is trained to utilize sensor data from disparatesources to determine a location of the user device 302. As used herein,the term “artificial intelligence” refers to any suitablecomputer-implemented artificial intelligence technique including machinelearning (supervised or unsupervised), natural language processing,machine perception, computer vision, affective computing, statisticallearning and classification (including use of hidden Markov models andBayesian network models), reinforcement learning including neuralnetworks, search algorithms and optimization algorithms (includingevolutionary computing) and automated reasoning. As an example, a neuralnetwork may be trained to receive the IMU data, thermal data, and aprevious location(s) data as input. This information may be used tooutput a corrected location of the user device 302 and/or to predict anext location where the user device 302 will be within a certain timeinterval.

In another embodiment, the sensor fusion module 312 may employ afixed-point algorithm, such as employing a Kalman filter. The Kalmanfilter may use the sensory input data from the thermopile array, theIMU, and previous location data, to build a prediction model. Theprediction model may take into account state variables (e.g., previouslocation(s), frequency of polling for thermal data, previousmeasurements of drift error, etc.).

FIG. 4 illustrates an example flow 400 for performing calibration of auser device 102 of the system 100 of FIG. 1, according to embodiments ofthe present disclosure. Although the operations are illustrated in aparticular order, some of the operations can be re-ordered or omitted.Also, some or all of the flow 400 (or any other flows described herein,or variations, and/or combinations thereof) may be performed under thecontrol of one or more computer systems configured with executableinstructions and may be implemented as code (e.g., executableinstructions, one or more computer programs, or one or moreapplications) executing collectively on one or more processors, byhardware or combinations thereof. The code may be stored on acomputer-readable storage medium, for example, in the form of a computerprogram including a plurality of instructions executable by one or moreprocessors. The computer-readable storage medium may be non-transitory.

In an example, the flow includes an operation 402, which involvespositioning thermal beacons 110-124 within a physical space 126, whichmay be a gaming environment. Using the example of a video game system100, a video game console 106 may prompt the user 102 (e.g., visually ona TV screen, audibly via a headset, or some other mechanism) to beginthe calibration process by positioning thermal beacons 110-124 in thephysical space 126 (e.g., game room). Depending on the type of gamingapplication, the video game console 106 may instruct the user on howmany thermal beacons should be used and what degree of coverage (e.g.,360 degrees, 180 degrees, etc.) is recommended. For example, if theapplication requires the user 102 to be able to move the video gamecontroller 104 (e.g., headset) in a 360 degree visual sweep of theperimeter of the room 126 while playing the game, then the user 102 maybe recommended to place thermal beacons around the perimeter of theroom, as depicted in FIG. 1. For example, the thermal beacons 110-124may be positioned both in the center of the four walls 112, 116, 120,124 and near the wall corners 110, 114, 118, 122. In this way, for agiven position of the headset 104 within the gaming environment, thethermopile array 218 of the headset 104 may be able to detect thermalsignals from at least two thermal beacons 204, 206 within its field ofview.

In an example, the flow includes an operation 404, which involvesreceiving an instruction that requests the user to move the user device104 to a first (or next) location. It should be understood that thisinstruction may come from directly from the user device 104 itself(e.g., an audio command from a headset), or from a computing hub 106(e.g., video game console) that is communicatively connected to the userdevice 104. In the case where the instruction may come from thecomputing hub 106, the user device 104 may iterate through a next step(e.g., operation 406, discussed below) in the calibration process andthen send a confirmation message (or information/error message) to thecomputing hub 106, which in turn may relay the next instruction messageto the user 102.

In the case where the user 102 is being instructed to move the userdevice 104 to a first location, and continuing with the gaming exampleabove, the headset 104 (e.g., via audio instruction, flashing light,etc.) may instruct the user 102 to move to the center of the room 128,which the user 102 intends to be the center of the gaming experience.The headset 104 may use this position 128 as the origin of the 3D modelthat the headset 104 builds and stores on the headset 104 as a result ofthe calibration process 400. It should be understood that the optimallocation of the origin may depend on the type of application, and, insome embodiments, may not be in the center of a room. In someembodiments, the user 102 may then signal to the video game console 106or the headset 104 that the user has completed the previous steps and isready to proceed.

In an example, the flow includes an operation 406, where the user device104 may determine a first (or next) location of the user device 104based on thermal data corresponding to thermal signals emitted by atleast two thermal beacons of a plurality of thermal beacons. In the casewhere the user device 104 is determining a first location, the user 102may already be in place, based on the previous operation 404. Continuingwith the gaming example above, the headset's calibration module 308 mayutilize thermal data received from the thermopile array 318 to determineat least two distance values, respectively, between the thermopile array318 and at least two beacons 204, 206 within the thermopile array's 318field of view. Based on this distance information, the calibrationmodule 308 may triangulate the headset's position and store thatposition and the distance values in memory 304. In the event that thecalibration module 308 is unable to detect at least two beacons withinthe field of view, it may prompt the user 102 with a warning (e.g., tocheck the positioning of the thermal beacons).

In some embodiments, at operation 408, once the calibration module 308has determined a first (or next) location based on information from atleast two thermal beacons within its field of view, the calibrationmodule 308 may determine if the calibration is complete.

In some embodiments, the calibration module 308 may perform thisdetermination 410 by determining if it has enough information toconstruct a 3D model of the 3D space. In other embodiments, thecalibration module 308 may pre-determine that it must capture all foursides of a room 126, and continually prompt the user to turn to a nextposition until it has capture data for all four sides. The calibrationmodule 308 may use any suitable mechanism to determine if thecalibration is completed. If the calibration is completed, then the flowmay proceed to operation 412, described further below.

If the calibration procedure has not been completed, then the flow mayloop back to operation 404, instructing the user to move to a nextlocation (e.g., position). Continuing with the gaming example above, theheadset 104 may output an audible instruction, tactile feedback, or anyother suitable method to indicate to the user 102 to move to the nextlocation. In one embodiment, the headset 104 may instruct the user to“turn in place approximately 90 degrees and then stop and wait forfurther instructions.” The calibration module 308 may then continue withthe calibration and perform operation 406, as discussed above. This loopmay continue until the calibration is completed.

In an example, at operation 412, the calibration module 308 maydetermine that it has enough data to construct a 3D model of thephysical space 126, with the first location serving as the origin of the3D model. Each of the determined distances (e.g., to the thermal beacons110-124) relative to the first location may be used to determine thedimensions of 3D model. In some embodiments, the calibration module 308may use any suitable mechanism to construct the 3D model. Once the 3Dmodel has been constructed, the calibration module 308 may store themodel on the user device 302 (e.g., in memory 304, or other storage).

FIG. 5 illustrates an example flow 500 for implementing SLAM on a userdevice that includes a thermopile array, according to embodiments of thepresent disclosure. In some embodiments, flow 500 may be performed afterthe calibration flow 400 of FIG. 4 is performed and a 3D model of the 3Dspace has been generated. Although the flow operations below discussutilizing a paddle as a user device (e.g., instead of a headset 104within the video game system of FIG. 1), any suitable user device and/orsystem may be used.

In an example, at operation 502, the user device may receive thermaldata corresponding to a thermal signal emitted from a thermal beacon ofa plurality of thermal beacons, the thermal signal being detected by thethermal sensor array of the user device. The user 102 may operate apaddle to play virtual ping-pong, whereby the game involves the usermoving the paddle into different positions as the user plays the game.The user 102 may also rotate the paddle and swing the paddle withdifferent rates of acceleration. As described above, depending in parton the type of application, the frequency at which the paddle may pollthe thermopile array 318 to receive thermal data may be increased toachieve higher location-tracking accuracy with respect to time. In someembodiments, this may involve the thermopile array algorithm module 314receiving a voltage reading (corresponding to a particular thermalbeacon's IR light signal) from the thermopile array at a frequency of atleast 120 Hz (e.g., receiving an update approximately every 8.33milliseconds).

In an example, at operation 504, the user device (e.g., paddle) maydetermine its location in 3D space, based on the data received from thethermopile array. Specifically, in some embodiments, for each voltagereading, the thermopile array algorithm module 312 may compute anassociated distance to a thermal beacon. As described above, the module314 may then triangulate its location using thermal signals from atleast two thermal beacons. In some embodiments, the module 312 may usethe 3D model of the 3D space within the gaming room 126 (previouslystored during the calibration process) to determine the location of thepaddle in the 3D space. In some embodiments, the paddle may utilize aconfidence value (e.g., generated by the thermopile array algorithmmodule 314) to determine its location. If, for example, at a particulartime, only one thermal beacon was detected, the confidence value may belower. As such, the paddle may determine to disregard that locationvalue, combine it with other location data to improve accuracy, orperform any other suitable behavior. It should be understood that, insome embodiments, the paddle may utilize only thermal data (apart fromother sensor data) to determine the paddle's location. However, in otherembodiments, the paddle may utilize, in addition to thermal data, othersensory input (e.g., IMU data) and/or variables to determine thepaddle's location, which may be further input into a sensory fusionalgorithm 312 for combining (as discussed in FIG. 6 below).

It should be understood that, contrasted with other SLAM applicationsthat may require computing external to the user device 104 to determinethe user device's location (e.g., external optical sensors, processingby a video game console, etc.), one technical advantage of the presentdisclosure is that it enables the user device 104 to determine itslocation using its own internal components (based in part on thermalsignals received from the thermal beacons). This may improve systemreliability, for example, in terms of reducing network connectivityissues such as latency, bandwidth limitations, etc.

In an example, at operation 506, and continuing with the example above,the paddle may transmit the location of the paddle to the video gameconsole 106. In some embodiments, the location data may be transmittedby the communications device 322 using any suitable communications path(e.g., WiFi), as discussed above. The video game console 106 may processthe location data using any suitable mechanism. In one example, thevideo game console 106 may use the location data to move an object beingdisplayed on the TV 108 (that corresponds to the paddle) to a newlocation, wherein the new location corresponds to the change of thepaddle's location in the 3D space of the gaming room 126.

In an example, at operation 508, the user device 104 may store its ownlocation on the user device 104. In some embodiments, previous locationdata may be in the form of X, Y, and Z coordinates that correspond tothe 3D model previously generated during calibration (see FIG. 4). Theprevious location data may be stored in memory 304 of the user device104. In some embodiments, the memory 304 may store a history of previouslocations, whereby a sensor fusion algorithm 312 of the user device 104(e.g., employing a machine learning algorithm or fixed-point algorithm)may use one or more of the historical location data points as input todetermine the user device's 104 current location. In yet otherembodiments, historical location data may be used by an algorithm topredict a future location.

FIG. 6 illustrates an example flow 600 for implementing SLAM on a userdevice that includes a thermopile array and an IMU, according toembodiments of the present disclosure. Similar to the flow 500 of FIG.5, flow 600 may be performed after the calibration flow 400 of FIG. 4 isperformed and a 3D model of the 3D space has been generated. Also,similar to flow 500, although the flow operations below discussutilizing a paddle as a user device 104 within the video game system ofFIG. 1, any suitable user device and/or system may be used. In someembodiments, flow 600 includes example operations that can beimplemented as sub-operations of the example flow 500.

In an example, at operation 602, similar to operation 502 of FIG. 5, theuser device may receive thermal data corresponding to a thermal signalemitted from a thermal beacon of a plurality of thermal beacons, thethermal signal being detected by the thermal sensor array of the userdevice. In some embodiments, this data may be further processed by thethermopile array algorithm module 314.

In an example, at operation 604, the user device may retrieve previouslocation data that was stored on the user device. For example, this maybe data that was stored by a previous operation 608. In someembodiments, previous location data may be in the form of X, Y, and Zcoordinates that correspond to the 3D model previously generated duringcalibration (see FIG. 4). In other embodiments, the previous locationdata may also include other data, including, but not limited to,inertial data (e.g., rotational rate, linear acceleration of the userdevice in 3D space at a point in time). This data may be received by oneor more sensor units devices of the user device 302 (e.g., IMU 320), asdiscussed further below.

In an example, at operation 606, the user device may receive IMU datafrom an IMU 320 of the user device. In some embodiments, the IMU datamay include acceleration data, orientation data, and/or rotation data ofthe user device in 3D space, as discussed in reference to IMU 214 ofFIG. 2. Accordingly, the IMU data may also be used to determine thelocation of the user device in 3D space at a point in time.

In an example, at operation 608, the user device may determine itslocation in 3D space by inputting thermal data (e.g., received atoperation 602), previous location data (e.g., received at operation604), and/or IMU data (e.g., received at operation 606) into a sensorfusion algorithm (which may correspond to the sensor fusion algorithmmodule 312 of FIG. 3). In some embodiments, other sensory input fromother sensors may be also used as input to the sensor fusion algorithm,including, but not limited to, a global positioning system (GPS)tracker. In some embodiments, each of the data received by the sensorfusion algorithm from operations 602 and 606 may correspond to datameasuring a location of the user device at substantially the same pointin time. However, in other embodiments, the data received by the sensorfusion algorithm from operations 602 and 606 may correspond to datameasuring a location of the user device at different time intervals. Inan example, and similar to as discussed above regarding operation 502,the thermopile array algorithm 314 may determine the user devicelocation based on thermal data from the thermopile array 318 at afrequency of 120 Hz (e.g., updating every 8.33 milliseconds). Incontrast and for example, a 20 Hz IMU may output an acceleration rateand rotational rate for a sample period representing the total motion ofthe IMU over 50 milliseconds. The sensor fusion algorithm 312 may usethe location data from derived from the thermopile array as a driftcorrection factor to correct for drift errors within the IMU locationdata. The sensor fusion algorithm may also utilize previous locationdata from operation 604 to increase the algorithm's accuracy, asdiscussed above.

In an example, at operation 610, and similar to operation 506 from FIG.5, the user device transmits its location that it determined inoperation 608 to the video game console.

In an example, at operation 612, and similar to operation 508 from FIG.5, the user device may store its own location on the user device.

FIG. 7 illustrates an example of a hardware system suitable forimplementing a computer system 700 in accordance with variousembodiments. The computer system 700 represents, for example, componentsof a video game system, a mobile user device, a proximity device, awearable gesture device, and/or a central computer. The computer system700 includes a central processing unit (CPU) 705 for running softwareapplications and optionally an operating system. The CPU 705 may be madeup of one or more homogeneous or heterogeneous processing cores. Memory710 stores applications and data for use by the CPU 705. Storage 715provides non-volatile storage and other computer readable media forapplications and data and may include fixed disk drives, removable diskdrives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, orother optical storage devices, as well as signal transmission andstorage media. User input devices 720 communicate user inputs from oneor more users to the computer system 700, examples of which may includekeyboards, mice, joysticks, touch pads, touch screens, still or videocameras, and/or microphones. Network interface 725 allows the computersystem 700 to communicate with other computer systems via an electroniccommunications network, and may include wired or wireless communicationover local area networks and wide area networks such as the Internet. Anaudio processor 755 is adapted to generate analog or digital audiooutput from instructions and/or data provided by the CPU 705, memory710, and/or storage 715. The components of computer system 700,including the CPU 705, memory 710, data storage 715, user input devices720, network interface 725, and audio processor 755 are connected viaone or more data buses 760.

A graphics subsystem 730 is further connected with the data bus 760 andthe components of the computer system 700. The graphics subsystem 730includes a graphics processing unit (GPU) 735 and graphics memory 740.The graphics memory 740 includes a display memory (e.g., a frame buffer)used for storing pixel data for each pixel of an output image. Thegraphics memory 740 can be integrated in the same device as the GPU 735,connected as a separate device with the GPU 735, and/or implementedwithin the memory 710. Pixel data can be provided to the graphics memory740 directly from the CPU 705. Alternatively, the CPU 705 provides theGPU 735 with data and/or instructions defining the desired outputimages, from which the GPU 735 generates the pixel data of one or moreoutput images. The data and/or instructions defining the desired outputimages can be stored in the memory 710 and/or graphics memory 740. In anembodiment, the GPU 735 includes 3D rendering capabilities forgenerating pixel data for output images from instructions and datadefining the geometry, lighting, shading, texturing, motion, and/orcamera parameters for a scene. The GPU 735 can further include one ormore programmable execution units capable of executing shader programs.

The graphics subsystem 730 periodically outputs pixel data for an imagefrom the graphics memory 740 to be displayed on the display device 750.The display device 750 can be any device capable of displaying visualinformation in response to a signal from the computer system 700,including CRT, LCD, plasma, and OLED displays. The computer system 700can provide the display device 750 with an analog or digital signal.

In accordance with various embodiments, the CPU 705 is one or moregeneral-purpose microprocessors having one or more processing cores.Further embodiments can be implemented using one or more CPUs 705 withmicroprocessor architectures specifically adapted for highly paralleland computationally intensive applications, such as media andinteractive entertainment applications.

The components of a system may be connected via a network, which may beany combination of the following: the Internet, an IP network, anintranet, a wide-area network (“WAN”), a local-area network (“LAN”), avirtual private network (“VPN”), the Public Switched Telephone Network(“PSTN”), or any other type of network supporting data communicationbetween devices described herein, in different embodiments. A networkmay include both wired and wireless connections, including opticallinks. Many other examples are possible and apparent to those skilled inthe art in light of this disclosure. In the discussion herein, a networkmay or may not be noted specifically.

In the foregoing specification, the invention is described withreference to specific embodiments thereof, but those skilled in the artwill recognize that the invention is not limited thereto. Variousfeatures and aspects of the above-described invention may be usedindividually or jointly. Further, the invention can be utilized in anynumber of environments and applications beyond those described hereinwithout departing from the broader spirit and scope of thespecification. The specification and drawings are, accordingly, to beregarded as illustrative rather than restrictive.

It should be noted that the methods, systems, and devices discussedabove are intended merely to be examples. It must be stressed thatvarious embodiments may omit, substitute, or add various procedures orcomponents as appropriate. For instance, it should be appreciated that,in alternative embodiments, the methods may be performed in an orderdifferent from that described, and that various steps may be added,omitted, or combined. Also, features described with respect to certainembodiments may be combined in various other embodiments. Differentaspects and elements of the embodiments may be combined in a similarmanner. Also, it should be emphasized that technology evolves and, thus,many of the elements are examples and should not be interpreted to limitthe scope of the invention.

Specific details are given in the description to provide a thoroughunderstanding of the embodiments. However, it will be understood by oneof ordinary skill in the art that the embodiments may be practicedwithout these specific details. For example, well-known circuits,processes, algorithms, structures, and techniques have been shownwithout unnecessary detail in order to avoid obscuring the embodiments.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flow diagram or block diagram. Although each maydescribe the operations as a sequential process, many of the operationscan be performed in parallel or concurrently. In addition, the order ofthe operations may be rearranged. A process may have additional stepsnot included in the figure.

Moreover, as disclosed herein, the term “memory” or “memory unit” mayrepresent one or more devices for storing data, including read-onlymemory (ROM), random access memory (RAM), magnetic RAM, core memory,magnetic disk storage mediums, optical storage mediums, flash memorydevices, or other computer-readable mediums for storing information. Theterm “computer-readable medium” includes, but is not limited to,portable or fixed storage devices, optical storage devices, wirelesschannels, a sim card, other smart cards, and various other mediumscapable of storing, containing, or carrying instructions or data.

Furthermore, embodiments may be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middleware,or microcode, the program code or code segments to perform the necessarytasks may be stored in a computer-readable medium such as a storagemedium. Processors may perform the necessary tasks.

Unless otherwise stated, all measurements, values, ratings, positions,magnitudes, sizes, and other specifications that are set forth in thisspecification, including in the claims that follow, are approximate, notexact. They are intended to have a reasonable range that is consistentwith the functions to which they relate and with what is customary inthe art to which they pertain. “About” includes within a tolerance of±0.01%, ±0.1%, ±1%, ±2%, ±3%, ±4%, ±5%, ±8%, ±10%, ±15%, ±20%, ±25%, oras otherwise known in the art. “Substantially” refers to more than 66%,75%, 80%, 90%, 95%, 99%, 99.9% or, depending on the context within whichthe term substantially appears, value otherwise as known in the art.

Having described several embodiments, it will be recognized by those ofskill in the art that various modifications, alternative constructions,and equivalents may be used without departing from the spirit of theinvention. For example, the above elements may merely be a component ofa larger system, wherein other rules may take precedence over orotherwise modify the application of the invention. Also, a number ofsteps may be undertaken before, during, or after the above elements areconsidered. Accordingly, the above description should not be taken aslimiting the scope of the invention.

What is claimed is:
 1. A user device comprising: a thermopile array; aninertial measurement unit (IMU); a processor; and a memory storinginstructions that, upon execution by the processor, cause the userdevice to: receive thermal data from the thermopile array, the thermaldata corresponding to a thermal signal emitted from a thermal beacon ofa plurality of thermal beacons and detected by the thermopile array;receive IMU data from the IMU, the IMU data comprising acceleration datacorresponding to an acceleration of the user device in athree-dimensional (3D) space and orientation data corresponding to arotational rate of the user device in the 3D space; determine a locationof the user device in the 3D space by inputting, to a sensor fusionalgorithm the thermal data, the IMU data, and previous location data;and transmit the location of the user device to a video game system. 2.The user device of claim 1, wherein the sensor fusion algorithm isconfigured to correct a drift associated with the IMU data.
 3. The userdevice of claim 1, wherein the execution of the instructions furthercause the user device to: estimate a first location of the user devicebased on the thermal data; and correct, by the sensor fusion algorithmand based on the first location, a drift associated with the IMU data.4. The user device of claim 1, wherein the sensor fusion algorithmcomprises an artificial intelligence (AI) model, wherein the AI model istrained to output a location estimate based on sensor data of differenttypes of position sensors.
 5. The user device of claim 1, furthercomprising a wireless signal-based position sensor, and whereindetermining the location further comprises inputting sensor data of thewireless signal-based position sensor to the sensor fusion algorithm. 6.The user device of claim 1, wherein the IMU data is generated at a firstfrequency, wherein the thermal data is generated at a second frequency,and wherein the first frequency is smaller than the second frequency. 7.The user device of claim 1, wherein the previous location datacorresponds to calibration data generated during a calibration procedureof the user device
 8. The user device of claim 1, wherein the previouslocation data comprises previous IMU data generated by the IMU.
 9. Theuser device of claim 1, wherein the execution of the instructionsfurther cause the user device to: estimate a first location of the userdevice based on the thermal data; and estimate a second location of theuser device based on the IMU data, wherein the sensor fusion algorithmoutputs the location of the user device based on the first location andthe second location.
 10. The user device of claim 9, wherein the firstlocation is associated with a confidence value, wherein the confidencevalue is based on a total number of thermal beacons that areconcurrently in a field of view of the thermopile array, and wherein thelocation is determined based on the confidence value.
 11. A methodimplemented on a user device coupled with a video game system, themethod comprising: generating, by a thermopile array of the user device,thermal data corresponding to a thermal signal emitted from a thermalbeacon of a plurality of thermal beacons; generating, by an inertialmeasurement unit (IMU) of the user device, IMU data that comprisesacceleration data corresponding to an acceleration of the user device ina three-dimensional (3D) space and orientation data corresponding to arotational rate of the user device in the 3D space; and determining alocation of the user device in the 3D space by inputting, to a sensorfusion algorithm the thermal data, the IMU data, and previous locationdata.
 12. The method of claim 11, wherein the sensor fusion algorithm isconfigured to correct a drift associated with the IMU data.
 13. Themethod of claim 11, further comprising: estimating a first location ofthe user device based on the thermal data; and correcting, by the sensorfusion algorithm and based on the first location, a drift associatedwith the IMU data.
 14. The method of claim 11, wherein the sensor fusionalgorithm comprises an artificial intelligence (AI) model, wherein theAI model is trained to output a location estimate based on sensor dataof different types of position sensors.
 15. The method of claim 11,wherein determining the location further comprises inputting sensor dataof a wireless signal-based position sensor of the user device to thesensor fusion algorithm.
 16. The method of claim 11, wherein the IMUdata is generated at a first frequency, wherein the thermal data isgenerated at a second frequency, and wherein the first frequency issmaller than the second frequency.
 17. A non-transitory computerreadable storage medium storing instructions that, upon execution on adevice, cause the device to perform operations comprising: generating,by a thermopile array of the user device, thermal data corresponding toa thermal signal emitted from a thermal beacon of a plurality of thermalbeacons; generating, by an inertial measurement unit (IMU) of the userdevice, IMU data that comprises acceleration data corresponding to anacceleration of the user device in a three-dimensional (3D) space andorientation data corresponding to a rotational rate of the user devicein the 3D space; and determining a location of the user device in the 3Dspace by inputting, to a sensor fusion algorithm the thermal data, theIMU data, and previous location data.
 18. The non-transitory computerreadable storage medium of claim 17, wherein the previous location datacorresponds to calibration data generated during a calibration procedureof the user device
 19. The non-transitory computer readable storagemedium of claim 17, wherein the previous location data comprisesprevious IMU data generated by the IMU.
 20. The non-transitory computerreadable storage medium of claim 17, wherein the operations furthercomprise: estimating a first location of the user device based on thethermal data; and estimating a second location of the user device basedon the IMU data, wherein the sensor fusion algorithm outputs thelocation of the user device based on the first location and the secondlocation.